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ARSTRACT 

, Measuring student writing skills can be done 
holistically, by ranking compositions without enumerating their 
linguistic, rhetorical, or information faatures, or atomistically, by 
viewing compositions as collections of these features. A variety of 
holistic approaches, including primary score testing, exist. Though 
their reliability has been questioned, they have content validity and 
can serve placement vurposes. Teacher, peer, and self-evaluation 
techniques’ can be applied to holistic approaches. At the other end of 
the holistic/atomistic continuum are tha widely us2d norm-referenced 
tests of editing skills, such as the General Fducation Development 
Writing Skills Test, Cooperative English Test, Missouri College 
English Test, McGraw-Hill Basic Skills System, Scot+-Foreman Test, 
Mills' Writing Pre-Test, and th? Test of Standard Written English. 
Critics of these tests cite immeasurable factors that lower the 
correlation of the quality 2f a whole essay with *he quality of its 
parts. Recen+ly developed procedures focus on the intellectual 
processes involved in writing, by measuring syntactic complexity, 
analyzing linguistic structures, and analyzing the degree of focus. 
Other recent developments exploit the capabilities of the computer in 
tabulating specific items such as mature word choice, natural 
language, and fluency. In addition to describing a pilot study of 
seven measures of writing ability, this paper describes and evaluates 
eich of the methods of measurement discussed. (AYC) 
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Measuring Writing Skill 


Introduction: Assessit or 


Atomistic approaches to writing view a composition as a collection 


Of features. In contrast, holistic approaches assume either that the 
in | i ne . 
= features are toO-numerous gr too complex to ‘assess separately or that 
the whole of the discourse has an.effect beyond the sum of its features. 
en, a _ ‘ 


-~Beeause that effect is the communicative purpose of the writing, it is 
ih = i A 

easy to argue that a human reaction is absolutely required for any valid 
judgment of writing quality. If the purpose of writing is to affect a 
reader's mind, there is no way to judge the writing without a reader's 
mind, Supplying this second mind for the writer to influence. is expen- 
sive and eurtabie, however. The reliability of graders has been 
Fecngnrzen ae a problem since the 1880's. ! Hitech contrasts the evalu- 
‘ation modes of Plato and Aristotle; he calls assessment "the single most 
important snag to practica! proyress in composition ceachang and 
eanearen.”” Thus it is worthwhile to consider both 


holistic and atomistic methods of assessing the quality of writing. 


 yaten C. Follman and James A. Anderson, “An Investigation of the 
Reliability of Five Procedures for Grading English Themes," RTE 1 (1966), 


190-200. . 


oe, D. Hirsch, Jr., The Philosophy of Composition, (Chicago: 


University of Chicago Press, 1977), p. 176. 


ges, 


Holistic. Scales 


Holistic evaluation of writing includes any method that sorts or 
rank orders compositions without enumerating their linguistic, rhetorical, 
or information fenburea,” One type of holistic evaluation matches the 
essays in question with a series of similar model compositions arranged 
according to quality. Analysis of the scale caaiameione ean show that 
the graders who arranged the scale had certain criteria in mind. For 


example, the London scale criteria for imaginative writing were realiza- 


. tion of the writer, comprehension of the audience, organization, density 


of information, and control of written language; and in the Sager Writing 
Scales, researchers, teachers, or junior high students assign zero to 

three points for each of these four aspects: vocabulary, elaboration, ° 
organization, and sieueare,” “Oietpuatys teachers can develop separate 
scales for different types of assignments. A dichotomous scale lists 
features to check as present or absent in the composition. An analytic 


scale allows a wider range of responses regarding each feature. It could 


“charles R. Cooper, "Holistic Evaluation of Writing" in ‘Evaluating 
Writing: Describing, Neasuring, Judging, ed. Charles R. Cooper and 


Lee Odell (Urbana:. NCTE, 1977). 


William T. Fagan, Charles R. Cooper, and Julie M. Jenson, 
Measures for Research and Evaluation in the "nglish Language Arts, 


(Urbana: ERIC and NCTE, 1975), p. 203. See also London Association for 


the Teaching of English, Assessing Compositions: A Discussion Pamphlet, 


(London: Blackie, 1945). 


include elaborate descriptions of each feature and definitions of high, 
middle, and low scores. Although raters doing general impression marking 
have no such list of specifics, their experience discussing sample papers 
in detail gives them a mental rubric to eran for their reliability. 
Other EapREoue holistic evaluation include center of gravity 
responses, which provide informal feedback to student compositions by 
summarizing them. This system helps students with revision but is not 
he intended for pearine.” Feature analysis and primary trait scoring depend 
on. narrower assessments of features relevant to specific assignments. 
Although holistic aus tuaeens are nokorteuuly uieatlabls, raters 
with siitlie hatiaeounas can be trained to achieve high. zeliabriity.° 
The training takes cooperation and time and thus adil to the expense of 


holistic evaluations, although Godshalk and Diederich assert that raters 


spending less than two minutes per paper can achieve reliability as high 


™ 


as .90 using some of the holistic scoring guides they describe. 


pana Elbow, Writing Without Teachers, (New York: Oxford Univ. 


Press, 1973), p. 86.° 


Op aut B. Diederich, Measuring Growth in English, (Urbana: NCTE, 
\ 
1974), Diederich established a scale by factor analysis. \ 


: "ered I. Godshalk, Frances Swineford, and William E. Coffman, 


The Measurement of Writing Ability, (New York: Educational Testing 
HY Service, CEEB, 1946). See also A.B. Myers, Carolyn McConville, and W.E. 
Coffman, "Simplex Structure in the Grading of* Essay Tests," Educat 


\ 
and Psychological Measurement, 26 (1966), 41-54. 


twenty minutes, (4) be capable of being completed on an 8 x 11" sheet of 


{ 
f 
a 
Mechanical correctness is usually part of the scoring guide and 
: | 
| 
prompt used in Los Angeles County Schools, Their holistic general 
impression marking assesses writing competence on a district-wide level, 
The prompts used for competency testing must meet a list of requirements: 
(1) be about real life situations, (2) not require students to ‘demonstrate 


knowledge outside of general experience common to the age and grade level 


of the test-takers, (3) generate a response that can be completed in 


paper, (5) allow for some creativity (for better responses), (6) require 
specific information, (7) be non-sexist in- name, nature, and antares and 
(8). be of interest to teachers, who are the graders. Los Angeles can get 
sufficient inter-rater reliability by training teachers for three hours 
to internalize a scoring guide based on the prompt. 

Los Angeles teachers have found they can teach many highschool \ 


classes to design a prompt and use it for peer grading. Preparing the . 


prompt helps the students to learn the requirements of good writing; peer | 
‘grading allows more frequent writing assignments and builds trust. If the 
teachers were of a mind to, they could incorporate ideas from Judine's 
collection or even dupptsate the ETS échnpat tion Seales by including in 

their prompt and scoring guide the results of Diederich's factor analysis: 
ideas and organization dau louwi ghted, wording, flavor, usage, punctuation, 


spelling, and handwriting. The major difference between holistic scales 


and primary trait scoring is that the latter might at times. be more narrow. 


8. 5. By : A ad As . 
Division of Curriculum and [Instructional Services (Julia M. 


Gottesman and Marian S. Schilling), A Common Ground for Assessing Competence: 


9 


College of the Mainland uses a descendent of the Cleveland 


: 9 
Rating Scale in its English 131 classes. A copy of it appears in the 
appendix, 


Tt includes percentage weights for organization, development, 


and coherence, and then subtracts points weighted for mechanical errors 
(grammar, punctuation, spelling). However, even experienced instructors 


need much more than the two minutes of the Educational Testing Service 
raters for mass grading. 


Holistic procedures have obvious content validity. They could 
serve placement purposes. 


When they involve rank ordering or specific 
scores, they can measure growth. 


Diagnostic purposes would require 


__————— 


rating with more care for detail. 


These elaborations increase the admin- | 
istrative difficulty, however. 


Primary Trait Scoring 


Although most holistic methods of assessment assume that the fea- 


| 
\ 
| 
\ 


« ° ¢ : 
tures of good writing are universal, primary trait scoring grows from a 


theoretical base that sees variation beyond the universal qualities, 


in Written Expression, (Downey, CA: Los Angeles County Supertinendent of 


Schools, 1978). See also Paul B. Diederich, "How ‘to Measure Growth in 


Writing Ability,” English Journal 55, (April, 1966), 435-49; Sister M. 


Judine, I.H.M., A Guide for Evaluating Student Composition, (Urbana, TL 
NCTE, 1965). 


af 
9 : 
Follman, op. cit., p. 193. 


. 


‘Considerable evidence exists that the mode of a composition (such as 


description, narration, exposition, or persuasion) affects the Way it 
is written. The effect extends even to ‘seemingly objective details of 


syntactic structure, 1° 


Wp avia P. Boder, "The Adjective-Verb Quotient: A Contribution 
to the Psychology of Language," Psychological Record 3 (1940), 310-343; | 
Ellen Frogner, "Problems of Sentence Structure in Pupils' Themes," 
English Journal 22 (1933), 742-749; J . C. Seegers, "Form of Discourse 
and Sentence Structure," Elementary fing list Review 10 (March, 1933), 
51-54;° Gerald Lloyd Kincaid, "Some Factors Affecting Variations in the 
Quality of Students' Writing," Diss., Michigan State University 1953, as 
described in Richard Braddock, Richard Lloyd-Jones, and Lowell Schoer, 
Research in Written Conpnsteion. (Champdizn: NCTE, 1963), pp. 83-95; 
Lois V. Johnson, "Children's Writing in Three Forms of euposttions” 
Elementary English 44 (1967), 265-269; Donald Richard Bortz, "The 
Written Language Patterns of Intermediate Grade Children When Writing 
Compositions in Three Forms: Descriptive, Expository, and Narrative," 
DAL 30 (1970), 5332 A (Lehigh); D. A. Nietzke, "The Influence of 
Composition Assignment Upon Grammatical Structure," DAI 32 (1972), 5476 A; 
Mike Pope, "The Syntax of Fourth Graders' Narrative and Explanatory | 
Speech," RTE 8 (1974), 219-227; Harold Rosen, "An Investigation of the 
Effects of Differentiated Writing Assignments on the Performance in 


English Composition of a Selected Group of 15/16 Year Old.Pupils," 


10 | 


Recent research analyzes the relationship of syntactic structure to the 


sex of the weiter? Variables of audience, topic, and tone or style 
have obvious effects. e 
‘The National Assessment of Educational Progress scored a mixture 
of 1969 and 1974 essays both holistically and descriptively and reported: 
"The nature of holistic scoring is such that one never knows precisely 
why a paper received the rating it did; the scorers themselves do not 
reflect on their decisions, only knowing that particular paper is 
better than some but not as: good as others. Consequently, this method 
of scoring provides a very reliable ranking of eeenia-se ranking most 
trained readers would endorse--but it tells us nothing about the sanece. 
except that some are better than others. Descriptive scoring of essays 
provides considerable information about essays, but it does not tell 
anyone how good a particular essay is when read. This study combines 
both systems in order to characterize specific writing... . The 
declines in holistic scores reveal as much about scorers as they do about 


students. Language is always changing, and scorers may prefer standards 


of written expression that are becoming outmoded. New standards are 


The Development of Writing Abilities (11-18), James Britton, Harold Rosen, 


et al. (London: Macmillan, 1975), pp. 2-3. 


tees P. Hiatt, "The Feminine Style: Theory and Fact," CCC 29 


(October, 1978), 222-226. 
} 


! 
e 


certainly different, but they may not be worse in any defensible senue,* 
Recognizing the differences, primary trait scoring looks at only 
.. . ’ the characteristics unique tommy specific assignment. The ‘primary trait 
score describes only the Bparenee aspects it was designed to assess. 
‘Usually an exercise earns one, point - ‘if the trait waa is absent, two 
points for its presences” three for competence, and four for axealionies”” 
It. requires careful preparation of the assignment and of the scoring 
guide. The theoretical base ae assumed variation limits its extension 
to other types of writing. Thus it is necessary to take great care in 
sbenandnk the assignment to make its application as broad as possible. 
e Extensive, careful preparation of a ee trait assignment and scoring 
guide was justified by its broad use in the National Assessment of Edu- | 
cational Progress, The use of Primary Trait Scoring for placement and 
dtaswwands is valid only where the design relates to such use. For = 


example, if we had a Piagetan developmental sequence of writing tasks, 


12 ieiting Mechanics, 1969-1974: A Capsule Description of Changes 


in Writing Mechanics, Writing Report No. 05-W-01 (Denver: National 
Assessment of Fducational Progress, 1975). See also Write/Rewrite: 
ey handenty and Persuasive Letter Writing, ieteding Report No. 05-W-03, 


49775. pp. 35 43. 


l3ina V. S. Mullis, The Primary Trait System for Scoring Writing 


Tasks, (Denver: National Assessment of Educational Progress, 1975), p. 6- 


es 


bes 


“h 


ak 


~ 


success on an assignment on one level might place the student within 


the Sequence. Thus: primary trait scoring might-also diagnose learning 


“needs within the sequence. Without such an assumed sequence, however, 


primary trait scoring is limited to measuring its specific aspect with- 
in the weight of the entire paper. Two similar tasks could trace growth 
for an individual teacher or for a national researcher. | 

Cultural bias that distorts responses becomes more likely as the 
assignment gains in specificity; yet specificity improves motivation 
and reliability of soe tna" In a single classroom assignment, extended 
‘breadth of application is not necessary. A teacher can use primary 
trait scoring. to*show students the streneths of their writing. The 
teacher seid vente write specific behavioral. objectives stating the 
ites traits. an assignment must show. Preparing a scoring guide pro- 
vides a detailed and exact description: of the performance expected on a 
specific rhetorical raat Primary trait scoring can bypare features of 
syntactic fluency, organization, diction, mechanical correctness, etc., 
to get a. aucstiogs searching the depth of thought. Andivate ‘have found 


that mechanics correlate with length and quality of developed Wdewa,”? 


Peet 13s chard Lloyd-Jones, "Primary Trait Scoring," Evaluating 


Writing, ed. Charles R. Cooper et al. (Urbana: NCTE, 1977, p. 42. 
Buenry B. Slotnik> and W. Todd Rogers, "Writing Errors: Impli- 


cations about Student Writers," RTE 7 (1973), 387-98. 


ig 
vw 


A primary trait scorer needs a firm grasp of the scoring guide 
to prevent the influence of unmeasured variables from decreasing 


reliability. Questions of judgment arise when a poor writer approaches 


the target without hitting it exactly. 


Peer Evaluation 


Who is capable of performing adequate holistic evaluations? 
Although the scaling methodologists of Educational Testing Service (ETS) 
stress the need for training wootats with similar backgrounds in order 
to achieve reliability, they do not state minimum competencies of the 


graders. 1° In 1976 E. D. Hirsch copyrighted "A Method for Forming Tests 


’ to Certify Assessors of Writing Ability" that requires candidates to 


assign grades to writing samples. The criteria is readability. A system, 
of quizzing audiences on the content of the samples determines the 
target etedee, Candidates able to assign target grades receive certifi- 
cation as assessors for writing done by a specific age group. Certified 
assessors can re create new tests, 2? 

Although certified assessment ability would certainly be useful 
for a teacher, good students also value this skill. In fact, all 
students of writing skills need an image of good writing, an image gained 


from reading. Beaven cites six assumptions that underlie student partici- 


on ederich, op. cit., and Godshalk, op. cit. 


ae 
Hirsch, op. cit. 


pation in writing dueeesiaubes”” Behind the assumptions are premises 


that growth in writing is a highly individualistic process related in 
many ways to growth in personality and procedures designed to foster 
such growth. One assumption is that growth in writing occurs slowly, 
which means changes are more noticeable at two or four year intervals 
than during the course of a semester or wiactee” Growth does not occur 
in isolation. Moffett believes that maturity alone will suffice for 
developing the cognitive structures of mature wviting.” Students need 
sensitization to themselves and their surroundings, smeaabes of their 
audience, and techniques for invention and revision. Maturation of 
writing skill takes goal-setting and risk-taking. Students need a 
climate of*’trust. 

These assumptions lead to ihdtvidueltced goal setting, which is 
especially useful at the beginning of the year within the single class- 
room. Self-evaluation grows from it. Students can .use checklists or 


any of the scales for holistic rating. They can evaluate the length of 


Mi aey H, Beaven, "Individualized Goal Setting, Self-Evaluation, 


and Peer Evaluation," Evaluating Writing, ed. Charles R..Cooper et al. 


(Urbana, NCTE, 1977), pp. 135-156. 


er aleer Loban, Language Development: Kindergarten through Grade 


Twelve, Research Report No. 18, (Urbana, NCTE: 1976). 


20 ames. A. Moffett, Teaching the Universe of Discourse (Boston: 


Houghton-Mifflin, 1968), pp. 76-83, 163. 


time they expended, their improvements, their strengths (which they may 


‘over look otherwise), and their weaknesses (which show that imperfection 
‘is acceptable and a source for learning). 

Peer evaluation has many of the advantages of self-evaluation and 
avoids the problem of excessive dependence on the teacher as a single 
audtence.”” It ruutieiente snuities writing to test its communicative 
function. Sager found that sixth graders could use a scale she developed 
to improve their writing as much as when only teachers evaluated student 
vork.”* 

Despite its values, peer evaluation presents problems. Many 
teachers themselves doubt its face validity, objectivity, and reliability. 
Administration is difficult: It takes extensive class time and usually 
a teacher trained in group dynamics. Although it could certainly offer 


useful hints on placement, diagnosis, and growth, it would be somewhat 


awkward. Its value for these purposes has not been tested and proven. 
Standardized Measurements of Editorial Skills 


Holistic evaluations are at the opposite end of the continuum 
from the widely-used norm-referenced tests of editing skills. An example 
of an atomistic test used only to certify competency is the "GED Writing 


Skills Test." The examinees need to recognize errors and make choices 


? . 
Pe pscaiaig Britton et al., The Development of Writing Abilities (11-8) 


(London: Macmillan, 1975). See also Beaven, op. cit. 


, 


9 


& 


Z ; 
Sager, op. cit. 
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of effectiveness. Skills include spelling, punctuation and capital- 
ization, grammar and usage, diction and style, sentence structure, and 
logic and jreduteatton’ 
The Cooperative English Test has broader uses: prediction, 
placement, assessment, and evaluation. It offers multiple choice 
questions on spelling, diction (such as double negatives), subject-verb 
agreement, comma splices, case, and use of semicolons and commas. On 
three-quarters of the questions, ‘students are to detect eivoeas’” The 
Cooperative Sequential Tests of Educational Progress (STEP) in Writing 
has students find errors or best choices regarding spelling, usage or 
diction, effectiveness, logic, punctuation, ate.” 
The Missouri College English Test is intended for both placement 
amd achievement measure, rere its authors recognize it can measure <% 


only a sample of "those aspects of learning commonly accepted as being 


important and gesauvabiac'e” In sixty items students hunt for errors of 


23cED Testing Program, Information Pamphlet on the New GED Tests, 


—— 


(Washington: American Council on Education, 1975). 


2S enopeative Test Division, Cooperative English Tests: Technical 


Report, (Princeton: Educational Testing Service,‘ 1960). 


23 Cooperative Test-Division, Cooperative Sequential Tests of 


Educational Progress, (Princeton: Educational Testing Service, 1957). 
lpabert Callis and Willoughby Johnson, Missouri College English 


Test Manual, (New York: Harcourt, 1965), p. 14. 


Pee Pd. 4 
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punctuation, capitalization, grammar, and spelling. In thirty other 
items students choose the best of four sentences or arrange given 
sentences in a paragraph. | 

The McGraw-Hill Basic Skills System writing test measures "skill 
in written communication" mainly for purposes of placement in college 
‘ English courses, Students find errors in capitalization, punctuation, 
and grammar; identify sentence types and fragments, and choose topic, 
developing, and rnensiudlne eentenane appropriate for given paragraphs. 
The publishers iad een correlate scores with srig criterion except their 
own texts. CTB/McGraw-Hill published also English test booklets with 
' separate sections on sounds, comprehension, mechanics, and organization 
of ideas. In the teak sedation choose items from lists construct 
outlines of the ioe iste of a sonpesttiane’” 

In 1978 Scott, Foresman published a multiple-choice diagnostic 
test to accompany its Handbook of Current English and Workbook of gaerene 
English. The Handbook states, "The- Diagnostic. Test is intended to test 


students’ mastery of grammatical and rhetorical skille."2? In fifty 


27 ace L. Raygor, ed., McGraw-Hill Basic Skills System Writing 


Manual, (Monterey, Calif: CTB/McGraw-Hill, 1970). 


2B rducational Skills Tests, College Edition, (Monterey, Calif: 


CTB/McGraw-Hill, 1971). 


2 | 
2 ite W. Corder, Handbook of Current English, (Glenview, Ill.: 


Scott, Foresman, 1978), p. xi. 


a 
Co 
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items, students select one correct sentence from four options; three of 
the choices have errors in grammar, punctuation, or spelling. In the 
thirty-five items of the rhetoric section, students: must choose the best 


topic, wording, sentence, or paragraph, Best means most effective or 


most appropriate for a 500-word theme.” Although this test can be is 


machine scored, its answers do not satisfy some English instructors any 
better than do the other editorial tests, a sample of which are listed 


here. Liberal and conservative approacaes result in differences here and. 


also in marking student’ writing directly," 


Mills has reported that her proofreading test called "Writing 
Pretest" is not as s00d a predictor of writing style as her "Sentence 
Pretest," sae had students follow examples analyzing pairs a¥ peneeuces 
and generating similar ones. >” She developed these tests for placement, | 


diagnosis, and measurement of growth in the text, Commanding Paragraphs, 


30 3 
Hugh H. Paschal, Diagnostic Test to Accompany Handbook of Current 


4 


a English and Workbook of Current English, (Glenview, Ill.: Scott, 


Foresman, 1978). 


§pichard Braddock, Research in Written Composition, (Champaign: 


“NCTE, 1963), p. 16. 


o2 elen Mills, "Language and Composition: Three Mastery Learning 


Courses in One Classroom,"" Journal of Basic Writing, Fall, 1976, she 


pp. 44 - 59, 


which requires tenth grade reading ability. Although a trained grader 


spends’ only five minutes on the "Sentence Pretest," Mills continues to 


use the "Writing Pretest" for its sdnindetrative ease. 

The College Entrance Examination Board "Sentences Test" consists 
of furty sentences. Each has a "no error" option and four spots of 
possible error in grammar, usage, word choice, idiom, capitalization, 

and punctuation. >> 

‘A more récent atomistic test of the editorial type is the Test 

of Standard Written English (TSWE), which Educational Testing oe ee 
published in 1974 to supplement the Scholastic Aptitude Test (SAT). Its 
fifty questions have éeudents either "recognize writing that does not 


{ td 


follow the conventions of standard written English" or "choose the best 


Be: a er, ' 
way of phrasing a sentence." — It tests grammar (such as ‘subject-verb 
agreement), usage, word choice, idiom, comparison,: coordination, and 
subordination, but it does not check on spelling, capitalization, formal 


grammatical terminology, or much punctuation. Some professionals ,have 
objected to this content of the TSwWE. >> 48 


aS college Entrance Examination Board, Comparative Guidance and 


Placement Program: Sentences, (Princeton: ETS, 1967). 


#7 pauedtional Testing Service, The Test of Standard Written English: 


A_Preliminary Report, (Princeton: ETS, 1975). 


7 ie dea Clark, "Problems with the Test of Standard Written 


_ English," CLAC - 4, (March, 1978), 10-13. 
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The publishers of TSWE show that it correlates with grades in 

college English courses and with writing performance. °° Their purpose 
for TSWE is placement only, not to certify achievement. Breland 
suggests that individual instructors who want to teach the kinds of 
things tested in TSWE should prepare their own posttests of progress.’ 
He warns -tuat teaching to vlan TSWE scores is difficult;. five points 
per semester is the maximum that can be expected. Teaching to raise 
the rating of an dduay is even more difficult. In this respect, these 
measures may resemble intelligence tests. 

Breland reports no significant differences in correlation for 
sex or mingrity status when all minorities are combined, but he recommends 
further aseabeh on separate minorities. For minority students and low- 


ability women, TSWE predictions tend to be higher than actual perfor-- 


mances; in addition, TSWE underestimates the actual. performance of high 


ability woneie” 


Suunter M. Breland, A Study of College English Placement and 


the TSWE, Project report 77-1 (Princeton: ETS, 1976). 


37 


——— 


Written English, Research Bulletin RB-77-15, (Princeton: ETS, 1977). 


& 


At Yale, students who scored low on TSWE wrote poor essays, 
but high scorers did not always write weld?” TSWE correlated best with 
subsentence and sentence parts of an English igapanttion rating. It 
correlated next best with SAT. verbal scores. The study concluded that 

. TSWE was adequate sed efficient for placement and more economical than 7 

an essay. It also revealed that justifiable objections to the quality 
of Yale student writing were related ye dvidness and the lack of logic 
or thought rather than.to mechanical requirements. 

Some users of TSWE see in it cultural bias and a discriminatory 
hidden agenda cide leads to overemphasis of conventional linguistic 


etiquette.” 


cee 


The Conference on College Composition snd Conmnteatien <6ece) 
has passed several resolutions on testing, one in 1974 stated: 
"Resolved, first, thar CCCC protest the inclusion of an objective. usage - 
test in the Scholastic Aptitude~Test, on the grounds that such tests 


are a measure of copyreading skill rather than a measure of student 


«as 


a8 i atth D. Hackman and Paula Johnson, Yale College Freshmen: 


How Well Do They Write? (New Haven: Yale Univ., 1976); see also by the 


@ . 
same authors, "Yale: How Well do Freshmen Write? Implications for 


' 


Placement and Pedagogy," College and University, 53 (Fall, 1977) 81-99. | 


“ oO etivah Funkhouser, "TSWE: ‘T is Not for Me," CLAC-3, 


(Nov., 1977), 13-16. 


a 
bys 
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ability-to use language effectively in connected discourse of their 

own composing; such tests place emphasis on mechanical matters of 

_ spelling, minetustton, and conventions of usage, rather than on clarity, 
‘appropriateness, and coherence of thought; such tests bond He discrimi- 
nate against minority students whose linguistic experiences often lead 
them to choose answers different from those expected by the test-makers;- 
and’ the inclusion of such a test may encourage secondary English teachers 
er toward the test at the expense of matters more fundamental to 
effective writing and sophisticated reading; and second, that CCCC 
encourage its members to resist the use of usage scores in the admission 


and placement of students"? 


nw hg : 
Pe a ma) 


In 1978, CCCC resolved that: "No student shall be given credit 
for a writing course, stared in a remedial writing course, exempted | 
from a required writing aciese, or certified for competency without 
‘submitting a piece of written discourse." The. resolution requires 
consideration of the complexities of both multiple choice and essay 


tests and further study of the entire issue of besting. 


‘isceulas tan No. 6" and "Resolution No. 7," CCC 25, (Oct., 1974), 


339. 


Tig enotutiton No, 1," CCC, (Oct., 1978), 309. 
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Braddock calls it charitable but inaccurate to’ say these 
objective tests measure editing ability; they ask for proofreading 
skills.“ They do correlate with success in courses well enough for 
screening when teachers can transfer students after reading their 
writing; bile what they evaluate is not actual writing ability. fifteen 
percent of the students with TSWE scores above the 95th percentile write 
essays that are below average. *> Test questions deal with the facets of 
English that are easiest to er Yet even here, cectiens arise and 
professionals disagree, Hooks points out.obvious errors, weaknesses, 
and a few strengths of objective baute, 

Tests tell students, teachers, and the public what the tester 
believes is — testing. If the content is patently trivial, the whole 
course is trivialized, demeaned, belittled, dehumanized. However, the 


College Board recognized its leadership role and issued guides for teachers 


42 Richard Braddock, "Evaluation of Writing Tests," Reviews of 
Selected Published Tests in English, ed. Alfred H. Grommon, Richard. 
Braddock, J. N. Hook, William A. Jenkins, Walter Loban, and Alan C. Purves, 

NCTE Committee to Review Standardized Tests, (Urbana: NCTE, 1976). 


"educational Testing Service, Guide to the Admissions Testing 


Program 1978-79, (Princeton:. CEEB, 1978), pp.9, 16. 
44 ask Force on Measurement and Evaluation in the Study of English, 

Alan Purves, Chairm Common Sense and Testing in English, (Urbana: NCTE, 1975). 
43 6. N. Hooks, “Tests on the English Language," Reviews of 

Selected Published Tests in English, ed. Alfred H. Grommon, (Urbana: NCTE, 


1976), pp. 76-117. 


planning advanced placement courses, directly influencing course content. 


‘The actual test is two-thirds essay: "the examination is unique because 
it requires writing as a direct measure of the students" ability to 
read and interpret literature and to use other forms of discourse 


effectively.""° = 


Subtotaling 


If it seems strange to call a direct writing test unique, the . 
complexities that justify it need explanation. There are logical reasons _ 
why evaluating the whole is more difficult than evaluating parts and 
summing them. These complicating factors lower the correlation of the 
quality of a ae essay with measurements of the quality of its parts, 
east as these mechanical tests just described and devices that will be . 
described in the following sections. The it eoaed problem is that what 
we need to measure is largely what Polanyi called Neacit knowledge," ae 
unmeasureables that underlie competence. 

Researchers have pointed at many complicating facsinee, Although 


correlation is not causation, in Loban's study proficiency in language . 


accompanied reasonable aericcwmect Ajay found that adding variables 


a 


educational Testing Service, Advanced Placement Course Description: , 


English Composition and Literature, (Princeton: CEEB, 1978),-p. 3. 


wh bate Ops ‘cits. 


° 


from content improves the prediction of weleing caskity from ‘style 
alone. ® Braddock alte claims from typewriter menufackupirs that. 
typists spell better and ‘write more than ethics students, Buk some 
research on this point is mikes” 

“Sanders concluded that timed impromptu conditions and acoiacad 
topics and modes limit both motivation and quality. Woodfin found 
that a longer eine allowance improves quality for third graders. In 
a 1963 description of the standard research models, Braddock warned, 
"Even if the investigator is srinarily interested in satine but gramar 
and mechanics, he should afford time fox writers to plan central idea, 
suanhin, and support as well as edit and proofread]. Otherwise their 
sentence Jesbouns and denotes will be, produced under artificial ; 


éiecinetances Furthermore, Diederich, Kincaid, and others conclude 


bie B. Ajay, Strategies for Content Analyses of Essays by Computer, 


Dissertation Abstracts International 34 (1973), 2375A. 


+? Braddock, op. cit., p. 51. 


' 


0c ara E. Sanders and John H. Littlefield, "Perhaps Test Essays Can 
Reflect Significant Improvement in peashnwn Composition: Report’ on a 
Micdeweei Attempt," RTE 9, (1975), 145. Footnote number 10 cites other 
evidence on the st iiann’ of mode. | 

se - Woodfin, “The Quality of Written Expression of Third 
Grade Children under Different Time Limits," Journal of Experimental 
Education 37, No. 3, (1969), 89-91. 


oF peddael, Research, p. 9. 
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that at least two writing samples are needed to allow students a chance © 


to do ‘their best.>> ; ane é 
eee “~ 


‘When we do bother to collect adequate writing samples, and ‘when eg 
we are able to take the time, we need valid, reliable, objective ways 
of assessing important: features. We need methods that satisfy Emig's 
complaint, "Most of the criteria by which students’ school-sponsored 
writing is evaluated concerns the accidents rather than the essences 
of Giketucebacee is waehitie, ase penmanship, and lengths 
rather than idhset development, chetorteal and syntactic sophistication, 
and fulfillment of intent." “Bhe found as she investigated how students 
actually wrote (in contrast to their teaching). that, for example, con- 


struction of a.formal out line did not correlate with highest quality 


“a? 


writing. Analvses of actual practices such as’ Emig and Christiansen ° 


have done lead to, changes in the content of composition eouraeas” 


Humanities as well as physics follows Heisenberg's principle: measure- 
ment influences and changes what is being measured. 
Mathematical methodology exists to cgunt the amount gf infor- 


mation in each word and sentence. © It requires determining previously 


33 incaid, op. cit., Diederich, 1974, p. 34. ~ ‘y 
4 ; 
Janet Emig, The Composing Processes of Twelfth Graders, , e 


(Urbana: NCTE, 1971), p. 93. 

*°prancis Christensen, Notes Toward a New Rhetoric, (New tones 
Harper and Row, 1967). | 
6.1 aus Weltner, The Measurement of Verbal Information in 


Psychology and Education, tr. Barbara M. Crook (New, York: Springer-. 


Verlag, 1973). : 
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known information in order to label new data. It must recognize that | 
the English language is more than a Markov series of left-to-right 
cecbabiitetea and that language has purposes beyond conveying infor- 
mation. Eventually HiG-aubmeny may have classroom use, but for now 


it concerns only researchers and planners of textbooks. 


Syntactic Complexity 


A method of measuring syntactic complexity of sentences has value 
for the researcher as well as for the classroom teacher who wants to 
determine readability or assess student writing. Several methods exist. 
Dupuis, Pox duanpte: presented a list of twenty-seven possible evand= 
formations: addition, deletion, reordering, combining, and waetaeionas”” 
Her inter-rater reliability was .94. Endicott proposed a scale of 
syntactic complexity based on the proportion of units, called co—memes, 
-in the surface structure of a sentence. His scale begins with a Sune 
tence lacking all the optional transformations except tds norphensac” 
Another test has students expand kernals of a iecibig of The Adventures 
of Huckleberry Finn. Analysis of their responses ‘reports the subordi~ 


nation ratio, words per subordi- 59 
nate clause and T-unit, and mean depth of embedding. This method and 


ee M. Dupuis, "Transformational Analysis of Compositions (TAC) ," 
Measures, ed. Fagan, pp. 193-94, ° 
58 


Anthony L. Endicott, "A Proposed Scale for Syntactic Complexity," 
RTE 7, (1973), 5-12. 
eritz Dauterman, "Syntactic Maturity Test for Narrative Writing,” 


described in Fagan, pp. 188-89, 


N 


other ways of analyzing syntactic complexity are based ou che ’wotk 
of Kellogg Hunt and Francis Christensen. Hunt showed that the length of 
the mean minimal terminable unit of a sentence (T-unit) increases with 
maturity.°” He thus provided a tool for describing syntactic fluency 
that has led to techniques of seadiane that use agitenvaccumbdniae 
activities. Another researcher, Dixon, compiled a list of indexes or 
predictors of syntactic maturity: mean T-unit length, words in final 
free modifier position, instances of final free modifiers, words in 
intra-T-unit coordinations, instances of such coordination, and total 
words and instances of free modifiers in all pudttdens. He found 
Hunt's T-unit the best single index, accounting for 44% of the distinc- 
tion in student writings. Free modifiers in all positions accounted 
for 30%; and a combination of T-unit and free modifiers can justify 58%. 
Much can be done with this type of aie. For eiaupla, Wolk had his 
students explore Christensen's hypothesis of the importance of final 
free modifiers by comparing their own essays with twelve professional 


information setictes.°* 


& 


60 ellogy W. Hunt, “Early Blooming and Late Blooming Syntactic 


Structures," Evaluating Writing, ed. Cooper, pp. 91-106. 


Ol award Dixon, "Syntactic Indexes and Student Writing Performance," 


Elementary English 49, (May, 1972), ,714-716. 
62. nthony Wolk, "The Relative Importance of the Final Free 


Modifier: A Quantitative Analysis," RTE 4, (1970), S9ff. 
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Measures of syntactic complexity demonstrate growth over long 
periods of time, several years. The question of the value of this 
complexity has a classic answer from Moffett, "Children's sentences 


63 Moffett also shows that 


must grow rank before they can be trimmed." 
vocabulary development, makes a single mature word substitute for a com- 
plex clause, and thus a non-superficial theory of complexity must 
‘consider semantics. 
Some research fails to show that longer T-units correlate with 

good wetting.” 

| Administration of measures of syntactic complexity are time 
consuming in actual counting, in obtaining an adequate sample, and in 
supiieatien, A warning was mentioned earlier that the mode of discourse’ 
affects the syntax and thus the length of tWated.”” One might hesitate 
with placement or diagnosis based Bale on T-unit length because of 
factors like context and effect, which influence syntax; yet there is no 
doubt that syntactic analysis provides a base for much good classroom 


instruction. 


OF th aaiee B. Moffett, Teaching the Universe of Discourse, (New York: 
Houghton Mifflin Company, 1968), p. 172. 

64, bert R. Potter, "Sentence Structure and Prose Quality," 
RIE, (eprine, 1967), 17ff. See also R. G. Martin, "A Prediction Formula 
for a Sample of 'Good' Writing," Dissertation Abstracts International 29, 
(1968), 1221A. Other studies also exist. See entire issue RTE 12 


(October, 1978), 3. 
65 


See footnote number f0. 
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Intellectual Processes 


a 


Maturation develops , child's Alii to abstract hierarchically. 
Moffett sees ''a parallel between qualifying ehoupht and elaborating: 
sentence sieusrures! as they develop éogettiee,”” Cognitive development 
over the years is more necessary than grammatical knowledge in geinolating 

_ Syntactic growth and elaboration, according to ghar after a longitudinal 
weaeys : | 

One of the early prophets of sentence-combining as a method for 
enhancing syntactic fluency was John Melldn; who looked at the pre- , 

requisite skills for sentence combining. He wanted syntactic maturity 

a statistical artifact and related increasing elaboration to conceptual 
davalovnent. If students lack cognitive development, he said, sentence 
combining adds only fog. ‘Dominant noun phrases cannot be more complex 
than the concept in the writer's mind. As the mineaies grows, so does 
syntactic memory. Organizing, chunking, Sestractie processes depend~on 


prior skills of memory and attention, which develop through maturation. 


6 
Bt cia, op. cit., pp. 35-36. Moffett, Ope Cit., p. 776 


OP oha C. Mellon, "Issues in the Theory and Practice of Sentence 


Combining: A Twenty-Year Perspective," in Senterce Combining and the 
Teaching of Writing, ed. Donald Daiker, Andrew Kerek, Max Morenberg 
(aeons, Univ. of Akron Dept. of English, forthcoming). See ee Mellon's 
earlier work, Transformational Sentence- ombining: A Method for Enhancing 


the Development.of Sentactic Fluency in English Composition, (Champaign: 
NCTE, 1969). | 


_ 


Y= 
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Piaget characterized the concrete operational student's thinking 


as, capable of decentration, of dynamic transformation, and of empirical 
reverses.°” These abilities are exactly what sentence combining requires: 
looking objectively at kernal data to re-arrange it effectively. 
Children's speech develops from social purposes through egocentric meanings 
to inner goals, according to Vraocaigs These stages resemble but do 
not gael parallel Kinneavy's aims for discourse: reference or expla-. 
nation (subject-oriented), persuasion (oriented to the audience), and 
Seas tecendion and literature (oriented to code or aedia).”* “However, 
Moffett showed how the modes of discourse kd in normal maturation. /7 
Another good exolamnetonet the process is Britton's.’> | 

Perhaps we need to look at intellectual processes in order to 


answer Ney's question: _How do sentence combining exercises affect the 


S  haubepk Ginsburg and Sylvia Opper, Piaget's Theory of Intel~ 
lectual Development (Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1969). 


OO ss Semenovich Vygotsky, Thought and Language, tr. and ed. 


E. Hanfmann and G. Vakar, (1934, Cambridge: MIT Press, 1962). 
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(Englewood Cliffs: Prentice-Hall, Inc., 1971). 


ce G1t.; 


el a Britton, Language and Learning (Middlesex, England: Penguin 


Books Ltd., 1970), pp. 203-237. 


psycholinguistic abilities of students so that observable changes 


appear in writing?!“ How does quality of writing relate to intellectual 
maturation, which develops through gradual increments, and oad does it 
relate to intellectual capacity, which traditionally .changes very little? 
If it is-as hard to raise a TSWE score as an IQ, it may be more to the 
point to teach and test intellectual processes directly, rather than 
indirectly,” The Yale study mentioned above found dullness and lack of 
logical thought a,more serious problem than mechanical contectueuns”. 
The English education theoretician Kinneavy has heretically questioned 
the competence of English teachers to teach and evaluate teeta!’ 
Aristotle taught invention; today we teach pre-writing. One 
approach to invention is Pike's tagmemic heuristic procedure. Pike rec~ 
ognizes the relationships in the hierarchy of linguistic elements. He . 


examines anything, linguistic or non-linguistic, as particle, wave, and 
Fete Ney, "Notes Toward a Psycholinguistic Model of the Writing 


Process," RTE (1974), 159-169. 


Seeatiad, A Study, pp. 50-51 


TO ackinais op. cit. 


” James Kinneavy, 'Sentence Combining in a Comprehensive Language 
Framework,"' in Sentence Combining and the Teaching of Writing,ed. 
Donald Daiker, Andrew Kerek, and Max Morenberg, (Akron: Univ. of Akron, 


forthcoming). 


a f . ; 
(wy 


30 


field, alt with ‘contrasts, variations, and distributions--a ninefold 
‘examination,’ © Although Pike developed some of the important ‘technical - 
procedures to analyze linguistic structures, he also stressed, "Beyond 
the linguist lies the artist."”? An ideal test would measure both 
linguistics and artistry. 

The most specific published method of analyzing intellectual 
processes. in student writing is ode11's,°° Odell develops focus, the 
Hct of the six processes, from Pike. Grammatical subjects reflect 
focus, the heirarchical level of abstraction. Second, connectors, 
comparative and superlative forms, negatives, negative affixes, and 
lexical items are linguistic cues to the use of contrast. The cues to 
classification and pattern recognition are predicate nominatives, labels 
of examples and instances, and. lexical terms of resemblance or class. 
Synonyms for change, become, begin, and top all indicate change, the 
fourth process. For placement in physical context the linguistic cues 
are nouns referring to geographical locations, objects in nhiyetedl. 


settings, or sensory properties of physical settings. To show time 

78 
-Discovery and Change, (New, York: Harcourt, Brace & World, Inc., 1970), 
oe 12% ‘ 
Mo annath L. Pike, "Beyond the Sentence," ccc15, (1964), 129-135. 


80) se Odell, "Measuring: Changes in Intellectual Processes," 


Evaluating Writing, ed. Charles Cooper and Lee Odell, (Urbana: NCTE, 


1977), pp. 105-132. 
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Richard F. Young, Alton L. Becker, and Kenneth L. Pike, Rhetoric: 


Sequence, English has adverbial elements noting existence before, during, 


or after a stated moment. Logical sequences use if...then constructions . 
and words implying a cause-effect relation, such as because, therefore, 
since, and consequently. - 

Odell recognizes that although counting individual cues is 
inaccurate, they direct attention to the processes that need tabulation. 
His independent seen achieved 88% agreement. Yet he wonders about a 
‘need for more subcategories and finer discriminations. He asks about a 
possible peaewnee of different subcategories or types of intellectual 
processes that develop at different ages. Such a sequence would parallel 
the syntactic development that Hunt found, axcuplivied by the betula 
flourishing of coordination. Odell aiteeets that identifying intellec- 
tual processes can distinguish quality of writing, diagnose problems, and 
measure growth. He considers intellectual analyses as complementary to 


analyses of syntax and of semantic choices. 
Computer-Aided Descriptions 
Computers can assist assessment by tabulating a variety of specific 
items. One area deals with appropriate mature word choice by comparing 
the writer's choices with standard frequency indexes.» A system by Finn 
requires typing student themes on IBM cards and programming the computer 


to print alphabetical lists of all the different words used in each theme 
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and their Ererusnnies” Finn points out that complicating factors 

make pehbanney aidLiionel lists of topic-imposed words and of words . 
that are rare fn writing but not mature (proper nouns, contractions, and 
slang, for example). Some common words appear equally. at different | 
grade levels and do not distinguish writers; they need no special 
attention. The list of remaining unclassified word choices may bring 
in irrelevant ideas. Finn suggests that teachers ‘could uae that list to 
analyze the coherence of the writing: 

One of the well-developed computer-assisted instructional programs 
is JOURNALISM. In it Bishop analyzes natural eaeieeee He relates 
the computer to spevitic exercises by supplying it with an instructor's 
list of key words, arranged in their best order. JOURNALISM does 
stylistic stuart by reporting variety in sentence length gol peewee: 
overuse of articles, ssleeieS adjectives, and adverbs. It checks 
spelling by printing any word not on a list of 17,000 words or a special 
topical supplement. In addition, JOURNALISM performs housekeeping duties 
by recording student progress. 

The work of Givlateanden; Hank; and Mellon underlies WORDS and 
COUNT, by Webb, programs that resemble Bishop's sonmnrr sit, Webb's 


COUNT reports on the number of words, sentence and paragraph lengths, 


81 : : i ; a oad 
Patrick J. Finn, "Computer-Aided Description of Mature Word 
Choices in Writing," Evaluating Writing, ed. Cooper (Urbana: NCTE, 1977), 


pp. 69-90. 
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2 : ‘ ; : 
Robert L. Bishop, "Computing in the Teaching 6f Journalistic 


Skills," On-Line 3, (May, 1974), no. 3, 5=1%. 


ranges, and deviations. WORDS compares the occurrence of thirty key 
sentence unit connectors and the occurrence of initial, medial, and 
final free modifiers with their frequencies in fifty random articles 
published in magazines in 1972.83 % 
Another approach is Slotnik's, Slotnik distinguished the ieasinad, 


uv 
a computer can make, called proxes, from what he calls trins, the 


. 


intrinsic qualities that interest human judas: For example, a computer 
can tabulate. the proxes of total number of words, the Ae recent words, 
the commas, and the function words. These proxes are ermoke. oe eiNeNeys 
and' fluency relates to the trin of quality of ideas and generalization. 
Mean and standard deviation of word size, head gerunds and past parti- 
ciples, and absence of uncommon misspellings relate to dtéeion in this 
plan. Colons, semicolons, quotation marks, and parentheses are uncommon 
punctuation marks related to the trin of Zoen ts thought and emphasis. 
Other trins measured are spelling, sentence structure, and pavepvaphatie. 
The validity of the ee needs proving. | 
A student with a printout from any of these computer programs | 


would have raw material to plan revisions, Printouts can suggest to 


a Jackson Webb, Computer Assisted Instruction Program in 


Writing, (Olympia, Wash: Applied Linguistics, The Evergreen State 


College, 1973). 


ae ene B. Slotnik, "Toward a Theory of Computer Essay-Grading," 


Journal of Educational Measurements 9, ~ (1972), 253-63. 


aes 


teachers the variables needing attention. Computer tabulations could 


also provide a, backup toa human evaluator, like a second reader. 
3 ; 
They have the reliability and the culture-fair quality of their program. 
* To measure growth in the specific aspects they count, they are ideals 
and an interpreter could use them to dinanoee some problens. Placement 
by computer is a long way off, even for specific aspects. Finn has not 
- yet vane even his vocabulary counts, and every topic assigned would 
need separate —S The content validity of the computer programs depends _ 


on their theoretical assumptions and on their uses. The biggest disad- 


vantage now is programming cost. Extensive use, however, might justify 


it. 


Cohesion 

Coherence is a quality desired in all writing. ‘It relates to both 
intellectual processes and to fluency with language, but it does not 
pe indicate logic and depth of chongtit: syntactic complexity, or 
correctness. Thus it is not complete as a measure bf wilting quality, but, 
probably like intelligence, it correlates. To count the gohesive ties in 
a piece of writing, I adapted a classification system developed by Halliday 
in 1973. Halliday: defined a cohesive tie as a signal sending the reader | 
to another sentence for its meaning. The test of a cohesive tie is 
Finding in the other sentence the ward or words to which it refers.’ As 
.used here, the term cohesion excludes Weta that relate the text to the 


_ real world; it also excludes the structural relations within sentences 


Me 


that give each well-formed sentence its own inner coherence, 


Fluent writers will be aware of their many options and be able 
to choose a variety of them, but perhaps eee much variety. The | 
fifteen types of sohediod include three types of iePiveneet “third 
person personal pronouns, deunenieatena (including the), and compara- ‘ = 
tives. Substitution and ellipsis can relate to a noun (by ones, same, | 
many, mine, more, etc.), to a verb (with do or by omitting’ tense, mood, 
or voice markers) , or toa clause (with SO Or as an answer to a aileeiaa), 
Conjunctions can add, contrast, continue, show cause, or relate to time 
andl winatoas The four lexical types of cohesion are repetitions of 
the same word; synonyms, hyponyms, ot related words; ceiee signcetens 
and general satis: (A more complete set of definitions and dedamtee 
appears in the appendix.) It would he rare to write coherently without 
at least lexical ties. t 
Counting ‘cohesive ties résembles counting intellectual processes 


with Odell's linguistic cues. Cohesion, however, may be more related to 


language fluency that to only intellectual process or depth of thoyght. 


‘In both counts, the range of types seems more important than the total 


of actual tokens. Within a short piece of writing (400 words), cohesion 


seems to have a ceiling of about seven to nine of the fifteen different 


8>michael A. K. Halliday and Ruqaiya Hasan, Cohesion in English, 


(London: Longman, 1976). 
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types. Poor writers use few of the options, but good writers do not 
_ use too much variety. 

Within the limits stated, 4 count of cohesive iss has some face 
validity, and it can measure growth. .Most students do use unre cohesive - 
ties at the end of a remedial writing course than they do at the be- 
ginning. a beaded reader couid count the cohesive ties quickly, but 
not faster than a two minute general impression marking. “White cohesive 
ties ae objective linguistic cues and may correlate highly with writing 


’ quality, .they are no more thau an index of it. The greatest value of a 


count of cohesive ties might be to supplement another incomplete measure, 


A Combination Measure 

Many schools use both a writing sample and an objective test 
on mect.anical and grammatical features. Cooper's definition of holistic 
marking excludes sewnadigees Pak enumerate linguistic, rhetorical, or " 
inronviel feutiven Thus we must sanadier sepaneealy Myklebust's 
Picture Story eer. Test. It has a purpose different from the tests 
described above. It diagnoses disorders of written language at a clinical 
level, although it has been used for placement in remedial college courses. 
.It consists of a stimulus picture for the student to write a story about, | 
_a eynicex aeeley. 8 productivity scale, and an abstract-concrete scale. 


The syntax score records addition, omission, substitution, and word order 


OS copery Op. cits; ps 4. 
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errors in punctuation, word usage, and word endings. It ignores 
erelstues’’ Myklebust seported, "it was. our eanetant feeling that the 
* A-C [Abst ract-Concrete] Scale mould be the most unreliable. To our 
amazement, this “is not at all the way it worked out Spaktistiaat ives 
the most troublesome scale was that of dyntan,” Scoring Myklebust's 
test takes at least fifteen minutes of secretarial time. (Instructors 
_auibbie more over alternate intentions for the mone: score.) Since 
dlternate forms a6 not exist, it cannot be used to measure growth. 
Although morms have been collected throughout:the world for children, 


ia do not extend into adulthood despite rapid changes expected be- 


— ages seventeen and twenty-one. 


* e 
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Appraisal testing seaaiiess of the Mainland includes the Nelson- 
Denny reading test and TSWE, a test that screens students for placement 
into regular transfer freshman English (Eng. 131) or into one of three 


remedial courses (Com. 11%, Com. 136, or Com. 139). At the beginning of 


‘ 
s 
n° fe 


these four courses, ‘all. students write an essay to verify the correctness 


a on ; a al 
of placement. They choose topics from a short list. Instructors of 


) . 


suetaur tlehusb: Development and Disorders of Written Language, 


Vol. I: Picture pe sey Language Test, (New York: Grune'& Stratton, ibaa 


*Suyklebust in an unput lished letter to Sallie Gordon Cowgill, 


January 6, 1972. 
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five of the communications classes and of two sections of transfer 

English class graciously provided unmarked copies of all the essays 

that their students wrote on two of the topics. The more popular topic 

turned out to be one that matched exactly a topic that Richard Lloyd- 

Jones had analyzed for primary trait scoring for the National Assessment 

of Educational Prepress.” The papers on that topic, the role of women, 

provided the base for a pilot study of ways of measuring writing quality. 
I checked the TSWE scores of the writers of each paper and 

_ selected seven nes whose writers had scored 20, 25, 29, 32, 36, 44, 

and 58, so that the samples would represent the range of TSWE scores 

(20 to 60 possible). Some papers had masculine names, and ‘one had a 

Spanish surname, reflecting the total college enrollment. I coded the 

papers,.cut the names off, and’ assessed them anonymously by six different 


measures and then used the code key to find the TSWE score and the 


reading level of the writer of each paper. Results appear in the 


\ 
\ 


appendix. - 

then one paper seemed to be on a combination topic I rejected it 
temporarily. It regained its position when it turned out to be the only 
paper in the Eee on the popular topic written by someone with a TSWE 
score of over.45. Apparently the other topic, asking preference for the 


good old historical times or the present, attracted all the students with 


- higher TSWE scores, although it was a less popular topic overall. 


er tougdones, op. cit. 
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For a holistic rating that would not influence the other 
analyses of these papers, two different instructors applied the College 
of the Mainland's English 131 "Composition Tabulation Sheet." It sub- 
tracts mechanical errors from a descendent of the Cleveland Rating 
edie A- copy appears in the appendix. One instructor had had ten 
years experience using: and modifying the scale for English courses. 
The other instructor had just begun teaching remedial writing. Their 
suvews were averaged. The total score represents a percentage for 
content and organization minus points for mechanical errors. 

For primary trait scoring I used only the final scoring guide 
_ printed with Lloyd-Jones's axederes Trying to use the scoring guide 
without sample papers for practice and discussion convinced me of the 
need for preparation to get valid assessment. I analyzed the appeals 
geeording to the scoring guide, but tabulation of those few results 
would be meaningless. Discussing them with the writers, however, would 
have been instructive. 

To get peer evaluation, I projected the papers to the lowest 
level remedied ‘eoukee and asked the students to grade them A, B, C, D, 
Ey oF fe-. They asked me to read aloud two of the papers of low legi- 
bility, eae of which they rated high and See Laws In the middle of the 
readings they paused to discuss criteria, but the siedeute had little 


in mind. Their discussion made no discernable difference in their 


beat trey Lot. 2304 


an, Clits. 


45 


40 


grading of the last papers. None received E or F, so I tabulated them 
on a four-point scale. No obvious consistency appeared. The mean 
and standard deviation for each essay appear in the appendix. 

To study syntactic complexity, I counted the mean T-unit length 
and also the number and proportion of ‘words in final free modi fier 
position., The results were higher than expectations based on narrative 
or expository writing norms. The assignment was argumentative,. however, 
and argumentative writing attracts syntactic Seuplevity.:* 

In listing intellectual processes, I noted that focus was always 
present as the grammatical subject of each sentence. Thus length’ 
directly influenced the count. I grouped and tabulated duplications. 
indicated by pronouns or synonyms, but I did not analyze the focus 
further. Contrast and time sequence or change seemed to overlap, but 
I tried to follow Odell's nivetesr™ It seemed worthwhile to report 
separately the temporal and logical sequence, adding one half for the 
latter. Most essays used most of the processes, but there was. a wider 
range in the total of subtypes of each process used, with duplicates 
“eliminated. The specifics number all the subtypes except those in focus. 
Computer programming for a small sample is impractical, but I 


did a count of the cohesive ties (types and items). See the appendix. 


ed footnote 10. 
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I selected scores on thirteen variables for computer analysis 


of correlation, matching them first to the English 131 total score 

and then to the TSWE score. Since the pilot sample is so small, it 

is hard to conclude. anything significant. Given the nature of the 
English 131 holistic total (which is content minus errors), part to 
whole correlation is expected. Reading correlated next best (.88) and 
then, perhaps surprisingly, cohesive types (.82). These should re) 
analyzed further in studies with a larger number of samples. The 
correlation of cohesive types is significant ~ .03. See the appendix 


for a "Table of Estimates of Usefulness of Different Measures." 
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APPENDIX 1 
COMPOSITION TABULATION SHEET 
: ENGLISH 131 


YES NO VALUE SCORE 


I. CONTENT AND. ORGANIZATION (100%) 
A. There is an appropriate 10% 
Introduction. 


B. There is a statedor strongly 
implied thesis idea that 
controls the entire paper. 15% 


C. The paper fits the assignment. 15% 
D. Each body paragraph in the paper 


has a topic sentence that supports 
the thesis statement and moves in a 


logical orderly direction. __ 15% 
—. Each body paragraph is adequately ‘15% 

developed to prove its topic 

sentence. 15% 


F. The paper includes a conclusion 
that smoothly ties the body of 
the paper back to the thesis. 10% 


G. Ideas in the paper flow smoothly 
from one paragraph and sentence 
to the other and there are no 


extraneous sentences or points; u 
i. e., the paper is free from : 
digressions. 10% 


._ H. Sentences show appropriate 
' co-ordination, subordination, i. 
etc., and are varied in structure. . 10% 


II. MECHANICS 
A. Spelling errors - ‘1 each 


B. Punctuation errors other than 
those included in run-on and 


fragments. 1 each 
Errors in the use of modifiers ° eo = 


other than dangling 
modifiers ; : 1 each 


4¢. 


D. ‘Dangling. modifier 3 each 


E. Errors in the use of 
pronouns i.e. agreements 
with antecedent, unnecessary 
shifts in person, case, 


vague antecedent. 3 each 
F. Errors in verb tense and 

agreement * 3.each 
G. Fragments run-on and comma ‘splice. 3 each 
H. Inadequate wording (words out of 

order, omissions, inaccuracy in usage) 1 each 

I. Nonparallel Structure 1 each 

J. Dull wording (careless repetition, 

"passive or form of to be when 
strong action verb is needed, etc.) leach 
‘ II. POSSIBLE POSITIVE POINTS . _ VALUE: SCORE 

A. Outstanding vocabulary (diction) 10%- 

B. ‘Imaginative approach: to subject 10% 

C. Depth of understanding of subject 10% 

D. Ability to analyze a problem in. 

an unusually perceptive or ; 
original way 10% 
FOOTNOTE: | 


The maximum number ‘of points that a student may earn on one theme is 100. 
Positive points are seldom earned and indicate a merit beyond what is 
normally found in freshman papers. 
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Intellectual Process Cues 


(from Odell, "Measuring Changes in Intellectual Prosesses," pp. 108-120 in 


Evaluating Writing by Cooper and Odell ) 


Process Cues 


Focus 


' Grammatical Subject of Each Clause 
Contrast 
Connectors (such as or, else, but, instead, though) 
= Comparative and superlative forms 
‘ Negatives (including without) 
Sanne verbs, adjectives, and adverbs stating contrast 
Classification 
| Predicate nominatives 
Example and instance labels or phrases 
7 Words meaning similar, resemble, class, and synonyms. 
Change 
a0 Words meaning change and its synonyms 
Verbs that could be rewritten with become 
Verb phrases that include a synonym for begin or stop plus a 
verbal | F 


Physical Context 


Nouns referring to geographical location, objects in physical 


setting, or sensory properties of a physical setting 


qo 
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Sequence in Time 
‘Hdverbial scien meaning before, during, or ‘after 
Logical Sequence 
Words implying cause-effect relationship (because, since); ‘the 


phrase if....then. 
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MEASUREMENT OF COHESION 


s ~ DEFINITIONS AND EXAMPLES OF TYPES OF COHESION 


REFERENCE 


1. Personal Pronoun: Third person personal pronouns:. he, him, his, 
she, her, hers, it, its, they, them, their, theirs. 


2. Demonstratives and The: a these, that, those, here, there, now 


Sa? SS Ee! SS? ee a? eee 


_ 3. Comparatives: Same, iennbaats similar (-ly), such; other, ' 
different else; more, less, as many, ordinal numbers; 
as & adjective, comparatives, (Superlatives are 
self- -defining and thus not cohesive.) 


4 


: ‘SUBSTITUTION AND ELLIPSIS 


=~ 1. Noun: Omission of noun or all or part of a noun group, usually - ; 
: represented by words such as mine,. this, his, hers; ‘ 
& any, each, many;. first, next, lasts one, two, more, 


a lots; best, better, good, superlatives or compara- - 
’ tives. Also substitution of one or ones (accom- 
panied by a defining modifier) or same me to teplace 


a count noun thing, nominalized process, attribute, 
or fact. 


2. Verb: Omission of a word or words from a complete verbal group, 
' including markers of tense, mood, or voice. Also 
substitution of do for the verb. 


! ° 4 
‘3. Clause: Omission of a clause structure, usually in response to a 
question that is asked,as response to.a statement, 
or in a reporting sequence. Also substitution of 
\S0, not, yes, or no for a clause. ‘ 


CONJUNCTION 


1. Adding: and, also, nor, or, or else, and not, and also, furthermore, 


add _ to “that, alternatively, Ey Ene by the way, incidentally; 
that is, in other words, e.g.3 thus; likewise, in ' 
the same way wav, on the ha otherhand, 


ee 
| er 


5. 


LEXICAL 


Ly 
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Contrasting: Yet, though, only, but, however, even so, all the same, 
in fact, actually; but, and, however, conversely; 
instead, on the contrary, at least, I mean, or 
rather, in any case, anyhow. 


Cause: So, then, therefore, eee on account of Eee 3 in 


rm le 


stances, “otherwise, under other circumstances; 
here, in this respect, apart from this. 


Time & Summation: Then, next, just then, before thats hitherto, in 
the | end; First, then, finally; at once, soon, 
next “time, next day, meanwhile, until then, at | 
this moment; 3 in conclusion, up to now, at. “at this” 
point, from now on; to sum up, to resume. 


Continuing: Now, of course, well, anyway, surely, after all. 

Same: Repetition of the same or closely related words, 
including inflections and derivations, not nec- 
essarily with the same meaning or referent. 

Synonym: Synonyms; near synonyms that are different but related 


members of the same subgroup, such as. Tuesday and 
Thursday or king and queen; hyponyms more _ 
specific than the original word; semantically 
related words expected to the found together and 
interpreted in terms of each other (tree and leaf). 


Higher Category: Superordinates; words of a more general, higher 
category that includes the original item as a 
subset of it. 


General Nouns: Generalized reference terms that could be replaced by 
a pronoun (R 1), usually accompanied by the 
reference item the or a demonstrative (R 2). They 
can carry an attitudinal meaning of adjective (the 
poor thing). People, persons man, woman, child, 
boy, girl, creature; thing, object; stuff; business, 


affair, matter; plage. question, idea idea. 
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UY 
MEASUREMENT OF COHESION: TABULATION 
IDENTIFICATION OF WRITING TO BE ANALYZED DATE 
Number of ‘Types Found: Cohesive Items Referent © Other Occurences 


No. of ‘No. 
TYPE OF COHESION . amie severe || Word (s) Sa No. of Sentence’ 


Reference 
1. Personal Pronoun | 


2. Demonstratives & a 
ee eee eae 


3. Comparatives 
Substitution & an 
‘1. Noun i 

2. Verb 

3. Clause 


Conjunction 
- Adding 


| 


1 

2. Contrasting 

3. Cause 

4, Time & Summar 

5. Continuing 
‘Lexical 

1, Same 

2. Synonym 

3. Higher Cates gor 


4. General Noun 


Number of sentences analyzed: w Goiedtd aie ‘Beginning 
Evaluation: ; Middle 
End 
Comment: _— 
Complete 
es 


re) = ’ Sr 


TABLE OF RESULTS 


i Paper [ 1 3 4 5 ‘ Mean 
Factor ; 
31) 34 45 61 72 85 


l. Holistic Scalé Total 29 
2, Student Ratings 
; S.D. 
.3. Intellectual: Process: 
Specifics 
Types 
Tokens _ a 
4. Cohesive Types 
5. Cohesive Items 
6. Final Free 
Modifiers: Percent . 
Words 
per T-Unit 


7. Length: Words 
T-Units 


8. Mean T-Unit Length 


i 
| 9. Primary Trait _ 
' Supports 


110. Holistic Content 
11. Reading Level 
12. Errors in Mechanics 


13. TSWE 


51.0 


2.8 


w 
o 


11.9 


ith 


Correlation W 


» | Holistic Scale 


° 
oO 


Total 


+68 


\ 
l. Holistic Total 


2. Student Rating 


- 3. Intellectual a 


Specifics 
4 Dohebtve Types 
5. Cohesive Items 
6. % Final Modifier 


._°7. Total Words 


8. Mean T.U. Length: 


9. P. Trait Support 
‘10. Holistic Content 
1. Rawdike Level 

12. Mechanic Errors 


13. TSWE 


CORRELATION © 


2 3 4 
5669 6067 8268 
1.0000 ° —_,9544 2651 
9544 “1.0000 ' 3019 
.2651 3019 1.0000 
.8075 . 6366 ~.0290 
1935 + «1300 . = +, 5862 
.6268 5167, 2238. 
-.0794 ~.0682 -.6222 
«2496 .1447- 4329 
.5601 .5898 .8257 
4856 .5207 .6733 
~.6315 - 37346 -.6511 
£2016 .2984 .7610 


a1. 


5 
«3505 


8075 


6366 
-.0290 
1.0000 

£4259 


5723 


“= .1859 
5164 


+3533 
No 

- 3075 
~.2267 


- 1. Holistic Total 
2. Student Rating 


3. Intellectual 
Specifics 


4. Cohesive Types 
"5. Cahesive Items 
6. % Final Modifier 
7. Total Words 

8. stegis T.U. Length 
9, P. ‘Trait Support 
10., Holistic Content 
ll. Reading Level 


12. Mechanic Errors 


13. TSWE © 


6 
-.4074 


.1935 


- 1300 


-.5862— 


4259 


1.90000 


e2127 


- 0634 


-.0802 


-. 3956 


-.6302 


~ 4849 


-.7019 


CORRELATION’ 


Matrix 
7, 
4309 


-6268 


5167 
.2238 
5723 
.2127 
1.0000 
0899 
£3795 
4894 
.1979 
-.0472 


. 3605 


8 


-.0794 


-.0682 


- 1859 
0634 
-0899 
1.0009 
-.6996 
-.5683 
-. 4006 
«2921 


- 0034 


- 6222 


52 


9 10 
.6868 9967 
2496 5601 
1447 5898 

4329 8257. 
5164 3533 

-.0802 -.3956 
. 3795 4894 

-.6996 -.5683 

1.9000 . 7026 
.7026 - 1.0000 
.5867 8673 

-.1432 ~.6974 
0374 6348 


12. 


136 


Holistic Total 
Student Rating 


Intellectual 
Specific 


Cohesive Types 
Cohesive Items 

% Final Modifier 
Total Words 
Mean T.U. Length 
P. Trait Support 
Holistic Content 
Reading Level 
Mechanic Errors 


TSWE 


CORRELATION 


Li 


.8880 


- 4856 


- 9207 
6733 
- 3075 
-.6302 
L979 
-. 4006 
- 5867 
8673 
1.0000 
-.8135 


Doe] 


Matrix 


py 


13 
-6160 


- 2016 


«2984 
. 7610 
-.2654 
-.7019 
3605 
-0034 
0374 
-6348 
09521 
-.5540 


1.0000 


a 


TABLE OF ESTIMATES OF USEFULNESS OF DIFFERENT MEASURES 


Admini- Content 
stration: Validity, 


Holistic 


MEANING OF POINTS 


Current Method 2 9 
Holistic 

Quickef Method 6 8 
Primary Trait 2 6 my 
Peer Rating 0 5 
TSWE 8 4 
Syntactic 

Complexity 2 5 
Intellectual 

Process | 2 6 
Computer 2 6 
Cohesion 6 7 
Myklebust 2 8 

Not applicable < 

1... Undesirable 
2 may. Not feasible 

3 | Weak 

4 Questionable, variable 

5 Unknown 

6 Incomplete, limited, partial use 
7 Possible 

8 Feasible 

g Good 


Placement: 


F3) 


Diagnosis: 


Growth: 
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Total Points: \ 


33 - 
28 
26 
16 


28 
25 


22 
24 
28 


34 
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